DiET in the context of MT evaluation

The DiET project is developing a comprehensive software package for the construction, annotation, customisation, and maintenance of structured reference data for the evaluation of NLP applications. The DiET system is implemented in a configurable, open client/server architecture with a central database system managing the data and a client integrating construction and annotation facilities and several servers supporting customisation procedures. The application of existing test data to new domains is supported by various means for customisation. These tools allow the user also to establish a relation between isolated, artificially constructed test items and specific corpora by means of a text profiling server, or to harmonise the vocabulary of the test items and the MT system under test by using lexical replacement functions. Since DiET provides an information system for handling different types of test data for a variety of applications, it can also be employed to support the (linguistic) evaluation of MT systems. Even though the DiET system will contain a substantial amount of annotated linguistic test suite data for English, French and German, the evaluators of an MT system will eventually be in charge of the construction of suitable material for their specific evaluation experiments. The DiET client offers system developers or professional evaluators the possibility to easily construct and annotate their own test data by either choosing the annotation types from the existing annotation type hierarchy or using the configuration mechanism to define new annotation schemata.